New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support #4695
Conversation
…support Assign location from the resp of insert() back to job.get()
… support Remove unused code
Codecov Report
@@ Coverage Diff @@
## master #4695 +/- ##
==========================================
+ Coverage 74.35% 74.39% +0.04%
==========================================
Files 430 430
Lines 27964 27967 +3
==========================================
+ Hits 20792 20807 +15
+ Misses 7172 7160 -12
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM according to the docs: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
Thanks for the review @Fokko |
My pleasure @ryanyuan |
… support (#4695) * [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support Assign location from the resp of insert() back to job.get() * [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support Remove unused code (cherry picked from commit 1c06c74)
… support (apache#4695) * [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support Assign location from the resp of insert() back to job.get() * [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support Remove unused code (cherry picked from commit 1c06c74)
… support (apache#4695) * [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support Assign location from the resp of insert() back to job.get() * [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support Remove unused code
… support (apache#4695) * [AIRFLOW-3874] mprove BigQueryHook.run_with_configuration's location support Assign location from the resp of insert() back to job.get() * [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's location support Remove unused code (cherry picked from commit 1c06c74)
BigQueryHook.run_with_configuration creates a new job and receives a response with location field. It should use that field for the keep_polling_job loop. Otherwise, it returns 404 job not found if the location is not US or EU.
After inserting new job and get the resp of the job details, it should retrieve the location and use it for the jobs.get request.
Make sure you have checked all steps below.
Jira
Description
Tests
test_bigquery_hook.py::TestBigQueryHookRunWithConfiguration
Commits
Documentation
Code Quality
flake8